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Preface 


This  report  summarizes  experience  in  maintaining  an  ex* 
perimental  computerized  information  and  reference  retrieval 
system  and  in  its  conversion  to  on-line  user  interaction  by 
remote  aocess.  The  Project  Director  for  establishing  the  docu¬ 
ment  base  and  the  system  design  was  Robert  S.  Taylor,  formerly 
Director,  Center  for  the  Information  Sciences,  Lehigh  Univer¬ 
sity,  and  currently  Director  of  Libraries,  Hampshire  College. 
Primary  contributors  to  the  system  development  and  operation 
were  Ronald  R.  Anderson,  Michael  A.  Jennings,  James  S.  Green 
and  Anthony  P.  Araico,  Research  Assistants  in  the  Center  for  the 
Information  Soienoes.  Dr.  William  A.  Smith,  Jr.,  formerly  Di¬ 
rector,  Computing  Laboratory  and  now  Associate  Director  for 
Systems,  Center  for  the  Information  Sciences,  contributed  to 
system  deeign  and  directed  completion  of  the  studies.  Exper¬ 
ience  from  this  and  related  studies  has  been  consolidated  into 
further  research  and  implementation  of  new  computing  systems, 
utilizing  the  established  document  base. 

Previous  reports  which  bear  directly  on  the  Office  of 
Naval  Reaearoh  studies  are: 

Experimental  Retrieval  Systems  Studies,  Report  No.  2, 
Systems  Manual  for  the  Experimental  Literature  Collec¬ 
tion  and  Reference  Retrieval  System.  Cervfcer  for  the 
Information  Sc TenoesT  R.  S.  Taylor,  R.  R.  Anderson, 

A,  P,  Amioo,  J.  S,  Oreen.  April  19o7. 

Experimental  Retrieval  Systems  Studies,  Report  No.  3» 

Part  1,  An  Associativity  Technique  for  Automatically 
Optimizing  Retrieval  Results"  Center  tov  the  Informa¬ 
tion  Sciences.  R.  H.  Anderson,  April  1967. 

An  Analysis  of  the  Factors  of  Conversion  from  a  Batch- 
Processing.  frape-Orlented  Retrieval  System  to  an  On- 
Line.  Real-Time  System^  MS  Thesis,  M.  A.  Jennings, 

Lehigh  University,  1^67 . 
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An  existing  experimental  document  and  reference  retrieval 
system  operating  in  batch-processing,  tape-oriented  mode  was 
converted  to  an  on-line  mode  with  user  interaction,  Objeotiveo 
were  faster  response  time,  integration  of  functions,  and  system 
accessibility  at  user  location.  Effects  on  file  organization, 
addressing  techniques,  file  maintenance,  communication,  schedul¬ 
ing  and  programming  are  analyzed.  Modification  of  an  existing 
batch  oriented  system  can  yield  an  effective  interactive  search 
capability.  Realization  of  full  benefits  of  the  on-line  con¬ 
cept,  particularly  maintenance  of  the  document  base,  requires 
system  redesign  rather  than  extension  of  existing  procedures. 

User  reaction  to  the  experimental  system  was  favorable  al¬ 
though  a  variety  of  improvements  were  suggested  for  an  opera¬ 
tional  environment.  Output  of  associativity  indicatox's  was 
valuable  in  reformulating  search  requests  or  in  adjusting  the 
volume  of  cited  references  for  both  experienced  and  unitiated 
users. 
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I.  Introduction 

Document  reference  Information  storage  and  retrieval  systems 
have  been  designed  and  developed  in  order  to  help  the  user  of 
such  a  system  in  his  information  seeking.  These  systems  in  their 
experimental  stages  have  been  generally  built  on  the  same  basic 
oonoepts  of  design,  structure,  and  system  goals.  The  environment 
of  these  systems  evolves  around  a  service  concept  which  provides 
document  referenced )  for  the  user  upon  his  search  reauest  to 
the  system.  The  document  reference  is  any  one  or  a  combination 
of  the  following: 

(a)  Document  title 

(b)  Author  of  the  document 

(c)  Location  of  the  document 

(d)  Publication  date 

(e)  Issuing  agency 

(f)  Location  of  the  agency 

(*)  Contract  and/or  grant  numbers 

(h)  Journal,  volume,  pages,  date 

(i)  Abstract  of  the  document 

To  provide  the  user  of  such  systems  with  a  more  easily  accessible 
and  expanded  service,  remote  access  and  machine-aided  cognition 
become  experimental  tools,  with  these  tools,  an  Intimate  collab¬ 
oration  between  a  human  user  and  a  computer  has  been  initiated. 

The  Reference  Retrieval  System 

The  system  upon  which  this  study  is  based  is  the  Experimental 
Reference  Retrieval  System  of  the  Center  for  the  Information 
Sciences,  Lehigh  University.  Two  major  objectives  in  maintaining 
this  system  are  to  provide  an  operating  collection  for  instruction 
and  experimentation  and  to  provide  a  basic  reference  system  for 
the  papers,  books,  and  reports  in  the  field  of  the  Information 
Sciences.  The  system  is  based  on  the  coordinate  index  concept. 

This  experimental  document  reference  and  retrieval  system 
includes  a  collection  of  over  2500  documents  related  to  the 
information  sciences.  Such  general  subject  areas  as  information 
retrieval,  library  automation,  management  information  systems. 
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artificial  intelligence,  computer-assisted  instruction,  logic, 
and  linguistics  are  included  in  the  collection.  All  the  docu¬ 
ments  are  housed  in  either  the  Center  for  Information  Sciences 
(CIS)  facilities  or  the  Lehigh  University  Library. 

The  selection  of  documents  for  inclusion  is  a  cooperative 
Judgment  by  the  CIS  personnel.  The  document  is  indexed,  using 
a  coordinate  indexing  approach,  by  one  or  more  graduate  students 
in  the  Information  Sciences  program.  The  index  terms  used  in 
describing  each  document  are  chosen  from  a  thesaurus  of  approx¬ 
imately  terms.  The  average  is  between  five  and  six  terms 
assigned  to  each  document.  The  index  terms  selected,  together 
with  the  remainder  of  the  data  used  for  document  representation, 
are  then  keypunched  on  IBM  cards  and  transferred  to  file  storage 
by  a  computer  program.  Two  seoarate  files  are  maintained, 
a  serial  file  and  an  inverted  file. 

Two  physically  separate  magnetic  tape  files  are  maintained 
to  support  the  operational  computer  programs  of  this  system  in 
a  batch  processing,  tape  oriented  (BPTO)  mode.  These  two  files 
are  the  serial  file  and  the  inverted  file. 

The  serial  file  is  linear  or  sequential  in  its  structure. 

The  file  is  alphabetical  by  author’s  last  name.  Two  records, 
sixty-four  computer  words  per  record,  are  used  for  each  document 
in  the  serial  file.  The  first  record  contains  author(s), 
title,  location  of  document,  document  number,  and  the  index 
term  numbers  for  that  document.  The  maximum  number  of  index 
terms  per  documont  is  twenty-five.  The  second  record  contains 
bibliographic  data  for  the  document.  This  is  the  issuing 
agency,  place,  publication  data,  pagination.  Journal  name, 
volume,  year,  and  contract  and/or  grant  numbers.  Figure  1 
gives  an  illustration  of  the  fixed  field  file  structure  of 
the  serial  file  records. 


Figure  l,  Serial  Pile  Organization 

The  inverted  file  is  maintained  in  ascending  order  bv  the 
unlaue  number  assigned  to  each  index  term.  Bach  record 
Includes  the  unique  term  number,  the  natural  lanruapre  represen¬ 
tation  of  the  term,  scope  notes  (apnroprlato  clarification  of 
index  terms),  a  usae;e  frequency  count,  and  the  accession 
numbers  of  the  documents  to  which  the  index  term  is  assigned. 

Two  types  of  search  procedures  are  available  to  users  of 
the  CIS  system,  both  usin*  term  numbers  for  searching  rather 
than  natural  laneuatre.  The  basic  search  program  allows  the 
user  to  formulate  a  search  specification  usinp-  any  combination 
of  from  one  to  four  index  terms  connected  by  the  Boolean 
operators  "and",  "or",  and  "not".  Three  options  are  available 
for  print-out  of  document  representations.  The  second  type 
of  procedure  is  an  associative  search  in  which  the  search 
specification  can  be  reformulated  by  inclusion  of  some  of  the 
most  highly  related  terms  in  the  table,  and  tried  astain. 
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Figure  2.  Inverted  File  Organisation 

A  significant  improvement  in  the  system  has  allowed  the  con¬ 
version  of  the  files  to  disk  storage,  allowing  a  direct  aooees 
rather  than  sequential,  form  of  processing.  The  file  contents 
and  format  remain  the  same.  Also,  on-line,  real  time  (OLRT) 
access  to  the  contents  of  the  files  is  provided  by  Model  33 
Teletypewriter  terminals  and  a  General  Electric  Datanet-15 
transmission  controller  on  a  dial-up  basis  to  a  OE  225  com¬ 
puter*  Conversion  to  this  on-line  capability  from  an  existing 
batch  oriented  operation  has  been  the  subjeot  of  Office  of 
Naval  Researoh  Contract  Nonr  610(06),  "Studies  in  On-Line  In¬ 
formation  and  Reference  Retrieval".  Thie  report  discusses  the 
effort  required,  the  benefits  achieved  and  the  user  reaotion 
based  on  experience  in  adapting  the  experimental  CIS  system  to 
remote  access  operation. 

The  system  is  completely  operational  and  is  described  in 
detail  in  the  form  of  a  systems  manual.!  At  the  present  time. 
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this  and  other  work  is  being  consolidated  under  a  separate 
NSF  grant,  "Prototype  Retrieval  System  Development  within  an 
Information  Resource  Lahore tory",  under  the  direction  of  Pro¬ 
fessor  D.  J.  Hillman,  The  experimental  literature  collection 
of  the  Center  Is  serving  as  a  data  base  for  the  study,  which 
will  utilise  improved  hardware  and  software  facilities  to  focus 
on  further  rese&roh  and  experimentation  emphasizing  on-line 
oontent  or  fact  retrieval. 


I I .  System  Characteristics 
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A.  The  Components  of  the  Tape-oriented  System 

The  computer  system  to  support  the  basic  document 
reference  and  retrieval  system  can  be  separated  into  three 
categories.  These  categories,  or  subsystems,  are  the  search 
subsystem,  the  update  subsystem,  and  the  listing  subsystem. 
These  subsystems  were  programmed  utilizing  FORTRAN  II  with 
the  G.E.  225  computer  system.  The  subsystems  are  a  series 
of  chained  programs  which  were  originally  developed  for  a 
batch  processing,  tape-oriented  (BPTO)  operation. 

The  Search  Subsystem 

To  execute  a  search  on  the  system,  a  search  state¬ 
ment  must  be  formalized  for  a  particular  query  using  the 
thesaurus  of  terms  and  the  Boolean  operations.  The 
thesaurus  terms  must  be  converted  to  their  respective  term 
numbers  and  olaced  with  a  Boolean  statement  in  a  fixed  field. 
For  example,  a  user  may  wish  to  search  for  documents 
dealing  with  computer  and  systems  but  not  programming. 

His  statement  would  then  look  like  the  following: 

computers  +  systems  -  programming 
In  the  statement,  +  is  the  Boolean  operator  and,  V  is  the 
or,  and  -  is  the  not  operation.  Before  this  statement  can 
be  submitted  to  the  system  via  a  punched  card,  the  respective 
term  numbers  must  be  substituted  as  in  the  following  example: 
(67500  +  88200)  -  (75500) 

The  system  will  accept  up  to  a  four-term  Boolean 
2 

search  statement.  When  the  search  statements  are  read 
into  the  search  program,  they  are  equated  against  the  serial 
file.  This  search  strategy  was  adopted  to  increase  search¬ 
ing  speeds.  Instead  of  searching  and  performing  the 
Boolean  operations  on  an  inverted  file,  a  method  of  searching 
only  a  linear  file  was  devised.  A  linear  file  organization 
for  tape-oriented  retrieval  systems  is  more  efficient  than 
an  inverted  file  organization.^  To  increase  searching  speed, 
the  serial  file  records  are  blocked  in  groups  of  eight. 
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A  second  type,  or  associative,  search  can  be  called 

by  the  user  which  will  display  a  table  of  terms  and  their 

associative  values  of  co-occurrence  with  the  terms  of  the 

query.  The  associative  values  are  computed  by  using  the 
a 

inverted  file. 

This  procedure  is  normally  used  when  one  of  two 
oases  arises:  (1)  the  searcher  wants  to  learn  something 
about  the  way  in  which  index  terms  are  used  in  the  system, 
or  (2)  the  searcher  has  already  tried  the  basic  search 
program  and  wants  to  either  increase  or  decrease  the  number 
of  documents  to  be  retrieved.  The  output  from  this  search 
procedure  is  designed  to  measure  the  relationship  between 
the  index  terms  used  in  the  search  specification  and  the 
other  index  terms  available  in  the  system. 

The  resulting  document  references  of  a  search 
request  are  displayed  on  a  printer.  The  number  of  searches 
run  at  a  given  time  can  be  "batched"  to  ten  search  statements, 
taking  only  one  pass  of  the  serial  file.  Figure  3 
illustrates  the  search  subsystem  in  a  simplified  flow  chart. 
The  Update  Subsystem 

If  a  system  is  designed  to  be  dynamic,  update  and 
maintenance  operations  must  be  Included  in  the  total 
eyatem.  The  update  operations  performed  on  the  files  are 
standard  basic  operations  (Figure  *0:  adding  new  documents, 
deleting  old  or  unwanted  documents,  adding  new  index  terms, 
and  deleting  index  terms. 

The  inputs  to  these  programs  are  a  variety  of 
different  punched  card  formats.  At  present,  150  new 
documents  can  be  added  during  one  update  run.  Deletion 
can  be  run  the  same  way  with  a  maximum  of  fifty-five. 

The  update  programs  must  run  in  a  two-phase 
operation  due  to  the  limitation  of  available  tape  drives. 

The  first  phase  operates  on  the  serial  file  only.  After 
the  first  phase,  the  tape  reels  are  changed  and  the  inverted 
file  la  mounted  for  the  running  of  the  second  phase. 

Obviously,  this  method  of  update  is  time-consuming  and 
consequently  costly  in  terms  of  computer  time  charges. 
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Because  the  programs  of  updating  are  costly  in  time  and  i 

money,  batch-processing  is  necessary. 

The  Listing  Subsystem  I 

A  series  of  listing  programs  are  included  to  j 

produce  a  book  catalog  of  the  documents  in  the  system.  1 

Additional  lists  of  the  files  are  used  as  a  verification  | 

to  check  update  operations. 

The  output  of  the  listing  programs  is  from  a  high  speed  S 

printer  (900  lines/minute).  An  illustration  of  this  sub¬ 
system  is  Figure  5. 

B.  Batch  Processing  Limitations 

A  number  of  general  problems  seem  inherent  in 
batch-processing  systems.  Some  of  the  problems  enumerated 
here  are  based  on  systems  work  connected  with  the  Lehigh 
system  and  with  other  systems  of  a  similar  nature. 

Flexibility:  Little  flexibility  is  available  in 
computer  processing.  Once  a  single  request  or  a 
batch  of  requests  are  submitted,  the  "die  is  cast". 

The  user  has  no  control  over  the  processing  of  his  ? 

request.  The  programming  associated  with  tape  systems 
is  generally  non-modular  in  structure.  Modifica¬ 
tion  of  the  system  programs  can  generate  lengthy 
changes  which  sometimes  result  in  program  redesign.'* 

Monitoring:  Program  execution  cannot  be  monitored.^ 

Programming  work  (debugging)  in  many  cases  becomes 

longer  than  the  initial  design  and  coding  of  the 

respective  program.  File  updating  is  not  guaranteed 

unless  a  listing  of  the  file  is  generated  after  a  j 

batched  update  run. 

Interpretation:  The  user  must  translate  a  request 

5  * 

into  the  language  or  system  code  of  a  fixed  format  j 

structure.  Where  requests  may  be  complex,  a  human  | 

intermediary  is  consulted  to  help  the  user  in  j 

transforming  his  request.  In  general,  the  more  J 

f 

intermediaries  interposed  between  the  user  and  the  ^ 

I  I 

!  I 

1  1  j 


Figure  5.  'Hie  Listing  Subsystem 


12 


document  reference  system,  the  greater  the  chance 
for  the  distortion  of  the  user's  original  intention 

7 

or  request. ' 

Turnaround  Time:  This  varies  for  a  particular  system, 
but  normally  this  time  is  not  positive  attribute 
of  a  batch  processing  reference  system.  The  user 
can  become  discouraged  if  his  request  is  placed 
in  a  long-  queue  for  processing.  For  the  Center 
for  Information  Sciences  BPTO  system,  turnaround 
time  varies  from  one  hour  to  three  hours.  This 

g 

is  low  compared  to  other  batch-processing  systems. 

In  summarizing  the  problems  connected  with  batch¬ 
processing  retrieval  systems,  Vincent  Oiuliano  states: 


Most  existing  computerized  document  retrieval 
systems  are  run  on  a  "closed  shop"  basis,  and 
there  is  little  or  no  facility  for  meaningful 
interaction  between  requestor  and  machine. 

The  requestor  usually  specifies  his  inquiry  bv 
writing  the  key  terms  on  a  special  form,  which 
is  later  keypunched  onto  punched  cards  and 
verified.  Later,  still,  the  cards  for  several 
inquiries  are  batched,  and  the  retrieval  is 
accomplished  by  sequential  computer  searching 
of  magnetic  tape  files.  The  answers  are 
recorded  on  magnetic  tape  which  is  not  printed 
until  after  the  searching  run,  and  the  answers 
are  finally  mailed  to  the  requestor.  Although 
the  computer  may  work  in  nanoseconds,  typical 
cycle  time  from  formulation  of  inquiry  to 
receipt  of  answers  maygbe  anything  from  several 
hours  to  several  days. 
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C.  The  On-Line  Concept 

On-line  computing  Is  a  concept  that  has  existed 
since  a  printer,  magnetic  tape  unit,  or  other  peripheral 
device  was  connected  directly  to  a  computer.  Only  within 
the  last  few  years  has  this  term  been  used  to  denote  a 
confjuter  system  that  employs  remote  terminals  or  teletype 
printers.  A  general  definition  of  on-line  is  as  follows: 
A  system  in  which  the  input  data  enter 
the  computer  from  their  point  of  origin. 

The  output  data  from  the  computer  are 
transmitted  directly  back  to  the  originating 
point* 10 

For  the  purposes  of  this  paper,  the  definition 
will  be  modified  to  clarify  the  on-line  concept  in  a 
reference  system  environment.  The  modified  definition 
is : 

A  document  reference  system  in  which  the  input 
query  enters  the  computer  from  a  remote 
(different  physical  locale  than  the  com¬ 
puter)  terminal.  The  reply  to  the  query  is 
transmitted  from  the  computer  directly 
back  to  the  same  remote  terminal. 

Simply,  the  transmission  controller  in  an  on¬ 
line  system  is  a  switch  which  allows  one  or  more  remote 
terminals  to  be  connected  directly  to  a  computer.  When 
utilization  of  the  computer  from  a  remote  terminal  is 
requested,  the  controller  after  "pollinp"  the  other 
channels,  selects  the  channel  desiring  service.  The 
remote  terminal  is  then  on-line  directly  to  the  computer. 
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In  this  particular  configuration,  no  other  remote  terminal 
can  gain  attention  until  service  is  completed  at  the  other 
terminal.  Figure  6  is  an  illustration  of  remote  terminals 
on-line  to  a  computer. 

The  remote  terminal  becomes  both  the  input  and 
output  device  in  such  a  reactive  system.  Entry  is  made 
directly  via  the  keyboard  on  the  terminal.  Outnut  is 
also  achieved  through  the  relatively  low  speed  printer 
which  is  adeauate  for  reactive  communication  but  not  for 
volume  printing. 


Figure  6.  On-Line  Remote  Terminals 
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XIX.  Transition  tp  On-Line  Retrieval11 
A.  Objective i  si  cpnv»r  eion 

In  the  following  discussion,  attention  is  focused  on 
the  advantage e  gained  in  converting  a  document  reference  system 
from  BPTO  to  an  GLUT  ayetem.  These  advantages  are  based  on  two 
assumptions t 

(a)  A  document  reference  system  is  a  service 
offered  to  its  user  community. 

(b)  The  service  system  should  be  as  user-oriented 
as  economically  feasible. 

Assumption  (b)  is  a  variable  that  can  be  determined  by 

analysis  of  an  Independent  system  environment.  The  uaer-oriented 
concept  of  a  service  system  has  a  catalyst,  the  OLRT  concept. 

The  user  is  severely  restricted  in  control  of  the  information 
retrieved  in  a  batch-processing  system  environment.  The  system 
procedures  and  formats  the  user  must  follow  establishes  the 
retrieval  interaction.  Consequently,  the  control  of  the  inform¬ 
ation  retrieved  is  a  function  of  the  system,  not  the  user. 

The  OLRT  system  returns  control  of  the  request  process 
to  the  user  through  a  user -computer  dialogue.  The  dialogue  gives 
the  user  the  feeling  that  he  controls  the  system.  He  is  free 
from  details  (procedures,  formats,  and  codes),  and  more  of  his 
attention  can  be  concentrated  on  formulating  search  requests. 

To  achieve  a  better  integration  of  user  and  system, 
some  general  objectives  must  be  established  in  the  OLRT  environ¬ 
ment.  These  objectives  must  give  the  relationship  a  more  mean¬ 
ingful  interaction,  as  well  as  lending  flexibility  to  a  document 
reference  system.  Characteristics  of  this  desired  flexibility 
are  growth,  flow  of  design,  and  experimentation. 

Centralisation 

The  objective  in  centralisation  is  to  place  large 
files  on-line  in  order  to  give  the  system  a  common  data  base. 
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The  centralisation  of  file a  ia  limited  by  the  potential  storage 
of  the  system  and,  normally,  haa  economic  he  well  as  physical 
boundaries. 

Faster  Response  Time 

It  is  obvious  that  for  conversational  mode,  faster 
system  response  is  necessary.  This  is  a  function  of  system 
design  in  both  hardware  and  software  configurations.  The  para¬ 
meters  involved  in  faster  system  response  are  file  structure, 
search  strategies,  queuing  algorithms,  and  hardware  access  times. 
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This  objective  is  based  on  the  processing  operations  of 
the  present  system.  The  batch-processed  tape-oriented  system 
has  associated  functions  that  must  be  performed  by  human  inter¬ 
vention.  The  OLRT  design  (when  properly  designed)  is  a  self- 
sustaining  system  whose  various  functions  are  integrated  and 
dependent  only  on  user  manipulative  operations.  In  contrast, 
the  human  operations  or  functions  in  a  tape-oriented  document 
reference  system  are: 

1,  Operation  of  computer  and  associated  equipment! 

(a)  initiating  every  system  run; 

(b)  changing  tape  reels; 

(c)  distribution  of  output  or  run  results; 

2.  Operation  involving  user-consulting  (system 
custodian) : 

(a)  teaching  new  users; 

(b)  helping  user  to  interpret  system  codes,  etc.; 

(c)  phrasing  user  request; 

(d)  submitting  a  user  request,  keypunching,  etc.; 

Along  with  the  BPTO  system  functions,  system  failure 

should  be  included.  If  a  failure  is  caused  by  equipment  mal¬ 
function,  the  system  is  "down".  No  searches  can  be  performed 
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until  the  failure  is  corrected.  An  OLRT  system  lends  itself 
a  degraded  mode  of  operation.  This  is  achieved  through 
fallback  and  automatic  recovery  procedures.  These  procedures 
allow  the  system  to  continue  service,  but,  generally,  result 
in  a  slower  response  mode  of  operation. 

System  Accessibility  at  the  User  Location 

The  remote  terminal  brings  the  system  to  the  user's 

location.  With  remode  location  access  to  the  system,  user 

communication  with  the  system  is  of  a  higher  degree  than  a 

batch-processing  system.  Richard  H.  Wilcox  states i 

Closely  related  to  simple  communications  is 
physically  convenient  access.  Experience  with 
conventional  computer  facilities  (batch-process¬ 
ing)  indicates  that  their  use  varies  roughly 
inversely  with  the  distance  from  the  user. 

System  accessibility  governs  a  system  "usage  function". 
The  resulting  increased  usage  is  a  safe  intuitive  concept,  but 
system  complexity  must  be  considered.  The  system,  depending  on 
its  service  function,  can  be  designed  for  general  user  simplicity 
or  the  design  could  be  focused  to  serve  a  highly  specialised 
user  community.  Associated  with  these  considerations  is 
whether  the  system  teaches  the  on-line  remote  user  or  whether 
a  human  intermediary  is  consulted  on  system  usage, 

B.  Sectors  gf  Conversion 

The  factors  of  conversion  are  those  major  functions  of 
a  total  system  that  are  affected  by  the  conversion  from  a  BPTO 
system  to  an  OLRT  system.  This  section  discusses  the  factors 
that  must  be  considered  in  converting  a  document  reference  and 
retrieval  system. 

1)  File  structures 

2)  File  addressing 

3)  Maintenance 

4)  Communication 
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5)  Scheduling 

6)  Programming  structures 

The  factors  are  discussed  from  the  view  of  experience 
based  on  systems  research  work  done  at  the  Center  for  the 
Information  Sciences.  First,  it  is  necessary  to  mention  briefly 
some  points  about  the  hardware  configuration  of  an  OLRT  system, 
for  the  hardware  has  a  direct  relationship  to  some  of  the 
factors  of  system  conversion. 

In  order  to  make  files  accessible  on-line  to  a  search 
system,  a  random  access  disk  storage  unit  is  necessary.  This 
type  of  unit  is  more  efficient  than  tape  drives  due  to  a  more 
direct  swans  of  file  addressing.  Another  piece  of  hardware 
peculiar  to  an  on-line  configuration  is  the  transmission  con¬ 
troller.  This  unit  is  a  computer  or  stored  program  multiplexer. 
It  a  purpose  is  to  maintain  communication  control  of  remote 
terminals  (i.e.  terminal  selection,  code  conversion,  etc.) 

This  multi-processing  mode  of  operation  frees  the  CPU  of 
operations  that  require  too  much  of  its  time.  Based  on  the 
hardware  - jquireoent s  discussed  above,  Figure  7  illustrates 
an  OLRT  hardware  configuration  for  a  document  reference 
service  environment. 

1.  File  Structures 

The  files  and  their  structures  are  the  foundation 
of  any  storage  and  retrieval  system.  These  structures  when 
properly  designed  yield  efficient  searching  and  storage 
operations  for  the  system.  They  also  give  the  system  a  degree 
of  flexibility  when  supporting  system  manipulations  that  can 
add  or  deists  file  records  without  program  modifications. 

Using  a  disk  unit  for  the  storage  of  the  files 
of  the  system,  a  different  search  strategy  is  used  from  the 
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Pigure  7#  An  OLRT  Hardware  Configuration 


one  diacusaed  for  the  tape-oriented  BPTO  system.  Disk  search 
is  performed  on  an  inverted  file  structure  while  serial  files 
are  used  subsequently  only  for  retrieving  author,  title,  and 
bibliographic  data. 

An  algorithm  is  used  to  determine  a  disk  address 
for  each  particular  document  reference  record  identified  by  a 
term  in  the  search  inquiry.  The  document  reference  record  ie 
in  a  directory  file  whose  structure  can  be  or  is  the  same  as 
the  serial  file.  This  inverted  file  search  strategy  is  a  more 
direct  means  of  accessing  records  on  a  disk  which  eliminates 
searching  of  a  complete  file.  The  access  or  addressing 
techniques  for  this  type  of  system  are  discussed  in  the  section 
on  File  Addressing  (below). 

Although  the  BFTO  system  operates  primarily  on  a 
serial  file,  maintaining  an  inverted  file  is  valuable  if  the 
possibility  exists  for  future  system  conversion.  Ix  en 
inverted  file  is  not  properly  maintained,  it  must  be  "created" 
from  the  existing  serial  file  at  time  of  conversion. 

Inverted  File  Structure 

The  inverted  file  structure  of  the  BPTO  system  for 
the  most  part  remains  the  same  when  converted  to  disk.  In 
order  to  maintain  storage  efficiency  on  a  disk,  transformations 
are  used  to  "tie"  disk  records  together  by  the  technique  of 
chaining.  Utilizing  a  chain  approach,  the  inverted  file 
records  do  not  have  to  be  stored  on  the  disk  in  sequence. 

One  approach  in  implementing  this  technique  is  to  allow  space 
in  the  inverted  file  structure  for  an  indicator.  This 
indicator  is  the  disk  storage  location  of  the  next  or  continued 
record  for  that  particular  term.  If  no  continued  record 
exists  beyond  the  present  record,  a  "flag",  or  termination 
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signal,  is  placed  in  the  storage  location. 

If  ths  physical  disk  record  consists  of  a  term  number 
beyond  the  record  siae  of  sixty-four  computer  words,  one  word 
is  ths  transformation  or  chain  indicator  and  the  other  sixty- 
three  words  axe  document  numbers.  The  last  logical  record 
containing  document  numbers  has  the  stop  "flag". 

Serial  File 

The  structure  of  the  BPTO  serial  file  can  remain  the 
same  on  disk  for-  all  practical  purposes  of  direct  conversion. 

If  Boolean  search  operations  are  performed  on  the  inverted 
file,  the  term  nuubers  for  each  document  do  not  have  to  be 
included.  The  fila/document  cam  be  compacted  with  other  biblio¬ 
graphic  data  or  additional  information  about  the  document. 

However,  if  associative  searching  is  used  in  the 
system,  compacting  must  be  disregarded  due  to  the  nature  of 
associative  searching  and  the  need  to  determine  co-occurrence 
of  terms  for  each  retrieved  document  reference. 

2,  File  Addressing 

The  previous  discussion  on  file  structures  in  am 
GLUT  system  described  the  use  of  an  inverted  file  structure 
for  searching.  Therefore,  in  the  conversion  of  the  BPTO 
system,  the  search  statement  remains  the  same  (Boolean  operators) 
but  ths  addressing  techniques  used  to  retrieve  the  file  data 
are  different. 

In  the  tape-oriented  system,  locating  was  the  process 
of  looking  at  each  document  in  the  file  serially.  Any  computer 
memory  location  or  disk  location  can  be  addressed  by  location. 

A  tape  can  be  addressed  by  searching  for  a  number  representing 
a  type  of  record  or  block  identification. 
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The  •canning  or  searching  of  a  tap*  atrial  file  ia 

not  a  direct  concept  of  addreaaing.  When  the  filea  are 

stored  on  disk  in  an  OLRT  system,  the  addressing  techniques 

14 

are  accomplished  by  five  storage  assignment  techniques: 

(a)  Direct  addressing 

(b)  Algorithm  calculation 

(c)  Scanning 

(d)  Table  look-up 

(e)  Randomizing 

Host  systems  will  use  a  combination  of  these  address¬ 
ing  methods.  Direct  addressing  is  the  easiest  to  implement  and 
most  often  is  most  economical.  Utilizing  the  table  look-up 
technique,  the  computer  searches  the  table  rather  than  the 
file.15 

The  other  three  techniques  were  impractical  or  pro¬ 
mised  to  be  time-consuming.  For  example,  the  algorithm  is  a 
mathematical  relation  that  generates  a  record  location  on 
disk  from  an  input  reference.  In  the  systems  under  discussion 
here,  the  input  reference  is  a  natural  language  term  or  terms. 
The  algorithm  could  not  differentiate  between  two  terms  that 
are  the  same  but  of  different  meaning,  such  as  manual  (by 
hand)  and  manual  (a  guide) .  Another  complexity  in  formulating 
an  algorithm  for  addressing  are  when  terms  of  different 
spelling  but  containing  the  same  letters  (i.e.  a  permutation) 
exist.  The  procedures  necessary  to  achieve  independence 
in  an  algorithm  become  too  great  for  practical  and  efficient 
implement  at  icn. 

Scanning  or  searching  the  files  is  a  lengthy  process. 
The  scanning  method  is  not  efficient  when  a  great  number  of 
files  are  maintained.  The  randomizing  technique  of  addressing 


23 


has  the  disadvantage  that  tha  correct  record  will  probably 
not  be  found  the  first  tine  when  addressing  the  file  .because 
of  likely  duplication  in  arrangement.^* 

The  two  Methods,  direct  addressing  and  table  look-up, 
when  used  together,  become  what  shall  be  here  called  a 
"storage  nap"  for  the  inverted  file.  This  approach  yields 
an  efficient  as  wall  as  a  flexible  technique  for  retrieve) 
of  documents, 

She  ftftr.Me  fiSE 

The  storage  nap  is  analogous  to  the  index  or  table 
of  contents  of  a  book.  It  conveys  to  a  searching  operation 
where  a  particular  record  is  located  on  the  disk.  For  a 
coordinate  indexing  systea,  the  index  terns  and  their 
respective  disk  locations  appear  in  a  table  structure 
(Figure  below) . 
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Figure  8.  Storage  Nap  Organisation 
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When  the  Boolean  search  statement  enters  the  system, 
it  is  inspected  for  the  operators.  Once  the  operators  for  the 
request  are  determined,  the  storage  map  is  linearly  searched 
for  the  respective  index  terms  of  the  statement.  When  a 
match  occurs,  the  operation  becomes  one  of  direct  addressing 
the  location  on  disk  of  the  first  record  for  the  index  term. 

If  file  updating  is  done  on  magnetic  tape,  tape-to- 
disk  routines  must  be  employed.  The  tape  updating  may  affect 
the  storage  map  disk  locations.  Each  update  of  the  files 
must  be  checked  against  the  storage  map  to  see  if  a  term 
record  has  been  added  or  deleted.  The  locations  of  the  term 
records  on  disk  do  not  have  to  be  sequential  but  when  file 
updating  on  magnetic  tape  is  used,  sequential  locations  are 
advisable. 

This  technique  of  using  a  storage  map  gives  a  system 
an  efficient  and  direct  method  of  accessing  di sk  records. 

The  elimination  of  searching  files  yields  faster  response 
times  to  a  request.  Other  information  which  might  be  stored 
in  this  table  rather  than  the  filO  itself  could  be  relevant 
during  retrieval  procedures,  such  as: 

(a)  Scope  notes:  Aids  or  additional  clarifi¬ 
cation  of  the  index  terms.  By  placing  them 
in  the  table,  additional  searching  is  not 
necessary  for  a  listing  of  the  system's  terms. 

(b)  Document  counter:  This  is  used  in  determining 
associativity  relationships. 

(c)  Last  location:  For  updating  purposes  (new 
terms,  etc.),  it  is  useful  to  know  storage 
restrictions  for  an  efficiently  packed  disk. 
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3.  Maintenance 

A  dynamic  document  reference  collection  requires 
that  techniques  for  file  maintenance  must  be  included.  The 
maintenance  function  for  file  updates  employs  basically  the 
same  logic  as  in  the  BPTO  system.  Procedures  for  updating 
change  when  updating  is  done  from  a  remote  console.  An  OLRT 
systea  whose  expected  service  "never  fails"  must  incorporate 
also  a  new  complete  set  of  special  purpose  utility  programs 
such  as  diagnostics  and  automatic  recovery  programs.  Most  of 
thsse  were  not  fully  implemented  because  of  the  experimental 
nature  of  the  system. 

4.  Coamunlcation 

When  a  document  reference  system  is  converted  to  an 
OLRT  mode  of  operation  with  remote  terminals,  one  of  the 
objectives  is  to  establish  as  natural  a  man-system  interface 
as  possible.  For  this,  a  natural  language  vocabulary  is 
needed.  Many  system  operations  must  be  put  in  the  form  of 
decisions  or  questions  for  the  user.  A  conversation  made  up 
of  questions  by  the  computer  and  answers  by  the  user  is  the 
result. 

With  real-time  or  conversational  response  environment, 
the  user  is  given  the  opportunity  to  change  his  mind,  to  think 
of  new  requests,  and  literally  to  control  the  system.  The 
ease  of  uee  combined  with  more  system  control  give  greater 
satisfaction  to  the  user. 

For  the  BPTO  system  discussed  in  this  study,  the 
request  or  input  from  the  user  is  a  Boolean  polynomial  search 
request  utilising  term  numbers.  If  the  user  can  submit  a 
request  with  natural  language  terms  (e.g.  automatic  +  indexing  - 
abstracting),  system  utilisation  for  the  user  can  be  made 
easier.  Tbs  user  in  this  latter  case  would  not  have  to 
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interpret  his  statement  in  specialized  term  code  numbers. 

Extending  this  notion  of  natural  language  requests,  a  natural 
language  request  sentence  may  be  allowed  (e.g.  "I  want  all 
documents  indexed  by  automatic  and  indexing  but  not  abstract¬ 
ing.").  This  kind  of  request  statement  requires  syntactical 
analysis  procedures  to  interpret  the  index  terms  as  well  as 
the  Boolean  opera:  'a. 

With  experience  in  establishing  an  on-line  teaching 
system  for  the  user,  a  query-respond  environment  may  be  allowed. 

This  environment  would  force  the  user  to  give  additional  infoap- 
mation  that  would  lead  the  system  to  a  more  relevent  answer. 

To  achieve  on-line  user  communication  with  an  operating  system, 
an  increase  of  programming  logic  and  programming  instructions 
is  required.  These  increases  are  determined  by  the  programming 
language  used. 

For  the  BPTO  system,  the  on-line  implementation  of  the 
search  programs  (associative  and  non-associative)  in  the  Lehigh 
system  resulted  in  an  increase  of  one  third  in  programming 
statements.  The  BPTO  search  program  contains  198  FORTRAN  II 
instruction  statements.  When  the  programs  were  converted  to  an 
on-line  conversational-mode  with  user  control  options  (choice 
of  searches,  output  wanted,  etc,),  programming  statements 
numbered  296.  This  allowed  the  user  to  respond  "yes"  or  "no" 
to  prompting  questions. 

5.  Scheduling 

Although  scheduling  is  a  programming  function,  it  must 
first  be  considered  independently  of  programming  structures. 

Scheduling  and  the  decisions  associated  with  algorithms  of 
scheduling  jure  matters  of  policy  and  form  the  core  of  any  OLRT 
system  operation.  The  formulation  of  decision  rules  for 
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handling  tha  eystam  when  mult iple-u ears  or  terminals  ara 
introducad  must  ba  avaluatad  and  worked  out  bafora  programing 
tha  functions.  Tha  Major  considerations  of  scheduling  ara  t 
(a)  User  priorities 

(1)  external 

(2)  internal 
(to)  Interrupts 

(c)  Queues 

(d)  Main  scheduling 

(a)  Input/output  scheduling 

These  ara  ail  issues  associated  with  tha  operating 
ey steal  of  sn  on-line  configuration.  Conversion  oust  provide 
for  control  of  these  matters  which  were  of  far  lass  importance 
in  a  hatch  processing  mode. 

6.  Program  Structures 

One  of  the  major  reasons  that  batch-processing 
programs  arm  not  flexible  in  their  structure  is  that  they  are 
often  not  segmented  into  their  logical  parts.  A  batch¬ 
processing  program  has  a  flow  of  procedures  and  operations 
that  terminate  in  some  final  result.  A  program  written  in  a 
straight  flow  mode  executes  its  instructions  to  accomplish 
one  complete  task.  That  is  ths  purpose  of  the  program.  Mhsn 
converting  to  sn  QLRT  system,  the  programs  have  to  be  structured 
for  flexibility  and  modification. 

Modular  Structures 

Modular  structuring  of  programming  procedures  and/or 
instructions  divide  the  total  program  into  its  logical  parts. 

A  logical  part  is  a  group  of  algorithms  or  instructions  that 
perform  one  or  mors  functions  on  ths  data  or  program  input. 

Thame  logical  parts  that  combine  to  form  a  total  program  are 
called  modules.  Xf  for  warn*  reason  s  module  or  logical  part 
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is  revised  or  eliminated,  other  modules  of  the  total  program 
should  not  be  affected.  In  an  operational  on-line  system 
these  modules  are  called  and  linked  at  compile  and/or  execute 
time  to  perform  tasks  within  the  system. 

In  a  dynamic  environment,  a  system  must  adapt  to 
change.  For  example,  a  system  may  normally  experience 
changes  due  to  experimentation,  policy  changes,  or  new  design 
strategies.  The  system  designers,  to  meet  this  changing 
environment,  must  create  programs  that  can  be  quickly  under¬ 
stood,  easily  modified  to  reflect  changing  conditions,  and 
whose  effect  on  related  programs  is  understood. 

The  construction  of  a  modular  main  program  in  an 
OLRT  system  uses  an  "overlay"  technique  and  provides  an 
efficient  way  for  programming  computer  core  memory  programs. 

It  conserves  memory  by  the  use  of  a  physical  part  of  memory 
for  more  than  one  modular  routine, 

C.  Programming  Considerations 

Although  an  on-line  interactive  mode  usually  requires 
additional  equipment,  programming  is  the  main  cost  of  system 
conversion.17  To  implement  the  BPTO  system  to  an  OLRT  system 
is  not  program  modification;  it  is  programming  systems  redesign, 
The  programming  system's  logic  changes  when  flexibility  and 
expanded  service  for  a  user-system  interface  are  implemented 
for  the  OLRT  environment.  Five  points  of  OLRT  programming 
must  be  considered. 

(1)  Computer  manufacturers  offer  standardized 
system  and  application  programs  with 
computing  and  communication.  These  programs 
must  be  modified  to  implement  a  particular 
system. 
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(2)  The  interaction  between  user  and  the  system 
must  be  supported  by  messages  to  guide, 
prompt  or  advise  the  user  in  his  own 
language  about  progress  of  the  search. 

(3)  Real-time  programming  must  have  interrupts, 
queuing,  scheduling  and,  diagnostic  techniques. 

(4)  Programs  must  become  modular .  System 
variables  cam  not  be  fixed  in  e  dynamic 
environment. 

(5)  Hardware  compatibility  is  sometimes  hard  to 
obtain.  For  example,  if  a  Model  33  teletype 
is  used  as  a  terminal,  s  codes  must  be 
translated  to  the  code  scheme  for  the 
particular  computer.  This  requires  a  sub¬ 
routine  or  modular  routine  of  a  table  look¬ 
up  nature. 

Before  programming  costs  can  be  estimated,  an  eval¬ 
uation  of  the  programming  personnel  available  must  be  reviewed. 
Progrananar s  can  make  a  system  conversion  costly  if  their 

work  is  not  evaluated  first.  Five  points  can  help  to 

18 

evaluate  present  or  future  manpower  output.  These  are: 

(1)  Experience  in  programming; 

(2)  Efficiency  in  programming; 

(3)  Creativity; 

(4)  Degree  of  training  needed; 

(5)  Compiler  languages  or  machine  languages  used. 
On  the  basis  of  these  points,  a  programming  output  for  a  staff 
may  be  estimated. 
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In  the  conversion  of  the  CIS  Document  Reference 
Retrieval  System,  emphasis  was  placed  upon  developing  the 
communication  with  the  user  in  the  se«.~^h  routines. 

Additional  initial  effort  was  placed  in  converting  both  the 
serial  and  invert#-  *  £i1»s  from  magnetic  tape  to  disk  form. 
Support  programs  to  i.  st  the  Thesaurus  from  the  inverted  disk 
file  and  new  author  listing  program.  During  the  period 
June  1966  to  April  1967,  three  graduate  research  assistants 
instrumental  in  development  of  the  batch  processing  system 
spent  about  half-time  each  in  developing  and  proving  these 
programs  and  in  relating  them  to  other  aspects  planned  for  the 
OLRT  capabilities. 

Features  included  in  the  search  program  were: 

(1)  Inclusion  of  an  associativity  table  print¬ 
out  option. 

(2)  Division  of  the  program  into  overlay  seg¬ 
ments  for  integration  into  an  anticipated 
Datanet  IS  Communication  Subcommittee  for 
the  GE  225. 

(3)  Reduction  of  line  feed  time  on  teletype 
printout. 

(4)  Feedback  of  the  equivalent  search  terms 
for  confirmation  by  the  user. 

Experience  with  this  program  indicates  the  following  advant¬ 
ages  of  the  disk  search  over  tape  search: 

(1)  More  natural  and  flexible  man-system  inter¬ 
action;  aided  by  entering  only  one  search 
at  a  time. 

(2)  Reduction  of  system  time  spent  in  actual 
searching  of  the  files  by  proportions  up 
to  100  to  1. 
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(3)  Greater  control  over  the  validity  of 
search  by  immediate  error  checks  and  feed¬ 
back  messages. 

(4)  Convenient  record  and  verification  of  the 
actual  descriptors  represented  by  the  term 
number  a  entered. 

(5)  The  member  or  absence  of  documents  yielded 
by  a  search  is  known  before  time  is  spent 
in  the  listing  of  results  phase. 

(6)  A  choice  of  three  options  for  output  format 
and  detail  is  offered. 

(7)  Printing  of  superfluous  blanks  in  document 
records  has  been  eliminated  in  output. 

(8)  Modification  and  reformulation  of  the 
search  request  based  upon  initial  results 
can  be  done  immediately. 

Utility  programs  to  achieve  file  maintenance  and  up¬ 
date  through  remote  access  veto  designed  during  the  period 
Kay-October  1967  in  order  to» 

(1)  Provide  means  for  convenient,  efficient, 
and  economic  display  and/or  modification  of 
the  data  base  file  from  on-line  terminals. 

(2)  Provide  a  complete  audit  trail  of  the 
additions,  deletions  and  alterations  made 
to  the  files. 

(3)  incorporate  flexibility  and  modularity 
to  facilitate 

(a)  changes  in  record  format  and 
file  organisation 
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(b)  addition  of  new  functions 
ic)  1  ..uependent  programming  and 
testing  of  software  modules. 

The  plan  called  for  functions  in  the  initial  phase  to  add, 
delete,  edit  and  display  individual  document  and  term 
records  and  to  add  or  delete  indexing  relationships. 

Estimates  of  the  programming  effort  to  develop 
these  data  management  functions  and  to  revise  the  GB  225 
monitor  to  include  the  Datanet  15  far  exceeded  the  amount 
of  personnel  time  available  for  the  remainder  of  the 
project.  These  features  were  basic  to  the  operational 
aspects  of  maintaining  the  system  and  would  have  lees 
effect  upon  the  search  user. 

The  system  was  experimental  and  was  to  be  phased 
into  other  work  and  the  GE  225  was  to  be  replaced.  There¬ 
fore,  these  latter  programs  were  conceived  but  not  imple¬ 
mented. 
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IV,  User  Reaotion 

A,  Study  Procedure 

To  assess  ths  user  reaction  to  the  on-line  system, 

&  structured  questionnaire  was  administered  to  a  selected 
group  of  subjects  after  using  the  Center  for  the  Information 
Sciences  system.  The  questionnaire  asked  the  subjects  to 
evaluate  the  system  performance  on  the  basis  of  three  major 
operating  level  components: 

(1)  System  search  procedures  -  vooabulary  oontrol, 
searoh  logic,  conversation  routine,  document 
collection,  indexing,  and  associativity. 

(2)  Hardware  facilities  -  terminal  and  communication 
line. 

(3)  Software  capabilities  -  access  to  system, 
response  time,  and  program  performance. 

The  reply  options  available  to  the  subjects  were:  favorably 
Impressed,  satisfactory,  needs  improvement,  severe  problem, 
and  no  opinion.  In  addition,  ample  space  was  provided  for 
comments.  A  supplement  was  issued  with  the  questionnaire 
presenting  suggestions  to  the  subjects  for  the  purpose  of 
assisting  in  the  formulation  of  their  evaluation  and 
generating  comments. 

To  eet&blish  a  natural  frame  of  reference,  thr 
questionnaire  was  structured  according  to  the  standard  search 
procedure  by  which  a  person  having  an  actual  information 
need  would  use  the  system.  The  procedures  included  were: 

(1)  Preparation  of  request. 

(2)  Submission  of  request. 

(3)  Initial  searoh. 

(10  Retrieval. 

(5)  Reformulation  of  request. 

(6)  Termination  of  search. 

The  subjeote  rated  the  components  of  the  system  for  each  of 
theae  procedures,  although  not  every  component  was  appli¬ 
cable  to  each  procedure.  A  sample  questionnaire  appears  as 
Appendix  II. 
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In  order  to  assist  evaluation,  the  study  was  con¬ 
ducted  by  grouping  the  participants  who  responded  to  the 
questionnaire.  Each  subject  was  given  a  retrieval  question 
representing  an  information  need.  Using  a  list  of  the  index 
terms  available  in  the  system  and  a  listing  of  the  contents 
of  the  inverted  file,  the  subjects  were  asked  to  formulate 
a  search  request  for  this  information  need  which  would 
retrieve  between  five  and  twenty  documents.  Once  the  subject 
had  formulated  his  request,  he  was  given  an  opportunity  to 
query  the  CIS  collection  using  the  on-line  system,  learn  the 
number  of  documents  identified  and,  if  necessary,  reformulate 
the  original  request  until  he  satisfied  the  conditions  of 
the  experiment.  No  assessment  of  document  relevance  was 
made;  instead,  the  subjects  were  told  to  assume  e  system 
was  operational  and  they  were  approaching  it  as  though  the 
query  represented  their  own  requirement  for  information. 

Replies  to  the  questionnaires  were  scored  on  the  basis 
of  the  following  scale: 

5  ■  favorably  impressed 
4  «  satisfactory 
3  *  no  opinion 
2  ■  needs  improvement 
1  -  severe  problem 

Each  subject1 8  evaluation  of  the  CIS  on-line 
system  was  calculated  from  this  assignment  of  values  and 
is  presented  as  Appendix  ill. 

Twelve  subjects  were  divided  equally  into  four 
group 8 ,  and  separate  queries  were  assigned  once  per  group. 

The  primary  purpo#»  of  the  study  was  to  expose  the  subject 
In  depth  to  the  features  of  the  system.  Therefore,  the 
queries  were  purposely  selected  such  that  there  was  a  high 
probability  that  a  subject  would  have  to  reformulate  his 
search  request  In  order  to  retrieve  between  five  and  twenty 
documents.  This  also  increased  the  likelihood  that  the 
associativity  search  procedure  would  be  used.  Of 
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secondary  interest  in  the  study  were  the  tiroes  required 
to  formulate  the  search  request  and  to  complete  success¬ 
fully  the  search  requirements. 

In  view  of  the  fact  that  all  of  the  subjects  were 
not  acquainted  with  document  reference  and  retrieval  systems, 
a  brief  period  of  orientation  preceded  each  experimental 
session.  An  oral  presentation,  approximately  forty-five 
minutes  in  length,  was  given  which  outlined  the  concepts  and 
techniques  employed  in  the  CIS  system.  This  was 
accompanied  by  an  on-line  demonstration  of  the  capabilities 
of  the  system.  In  addition,  an  introductory  supplement, 
detailing  the  functions  of  document  reference  and  retrieval 
systems  In  general,  and  the  CIS  system  In  particular,  was 
distributed  one  week  before  the  experimental  session  was 
conducted. 

B.  Selection  of  Subjects 

Two  factors  were  considered  in  the  selection  of 
the  twelve  participants:  (1)  the  subject's  knowledge  of 
document  reference  and  retrieval  systems,  and  (2)  the 
subject's  experience  with  on-line  systems.  Two  levels  of 
each  factor  were  used,  making  a  total  of  four  unique  groups 
with  three  subjects  in  each  group.  The  characteristics  of 
each  group  are  shown  in  Figure  9. 

Knowledge  of  Document  Retrieval  Systems 


On-Line 

System 

Experience 


High 

Low 

High 

I 

II 

Low 

III 

IV 

Figure  9.  Experimental  Subject  Groups 
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Subjects  from  Groups  I  and  III  were  selected  from 
eligible  graduate  stulents  in  the  Information  Sciences 
program  who  had  not  been  Involved  with  the  design  and  imple¬ 
mentation  of  the  CIS  system.  In  all  cases,  the  on-line 
experience  of  Group  I  subjects  was  obtained  from  using  the 
CIS  system  in  addition  to  other  on-line  information  systems. 
Group  III  members  had  seen  the  CIS  on-line  system 
demonstrated,  but  had  never  used  it  themselves.  Group  II 
and  IV  participants  were  selected  from  volunteers  In  a 
graduate  level  course  in  industrial  information  systems. 

The  on-line  experience  of  Group  II  members  had  been  obtained 
primarily  from  contact  with  time  sharing  and  communication 
systems . 

C.  Results 

For  purposes  of  analysis,  evaluations  from  the 
appropriate  groups  were  combined  according  to  the  experience 
factors  under  consideration,  and  are  presented  below  In 
tabular  form.  Table  I  compares  the  levels  of  experience 
with  on-line  systems  and  Table  II  shows  the  results  obtained 
when  the  two  levels  of  knowledge  of  document  reference  and 
retrieval  systems  are  compared.  Evaluations  for  each  of  the 
six  phases  of  the  search  procedure  are  presented,  together 
with  an  average  for  the  entire  system. 
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Table  I.  Experience  with  On-Line  Systems 
(Groups  I  and  II  High) 
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Table  XI.  Knowledge  of  Document  Reference 
and  Retrieval  Systems 
(Groups  X  and  XXX  High) 


The  average  times  required  for  request  formulation 
and  the  on-line  search  procedure  during  the  experiment  are 
grouped  according  to  the  two  factors  under  consideration, 
and  are  shown  in  Table  III.  A  detailed  breakdown  of  these 
times  for  each  of  the  twelve  subjects  is  given  as  Appendix 
III. 
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Table  III.  Times  for  Request 

Formulation  and  On-Line  Search 

(Minutes) 

Other  experimental  results  of  interest,  include 
the  actual  requests  formulated  by  the  subjects  and  the 
comments  given  in  response  to  the  questionnaire.  Both  of 
these  items  were  considered  to  be  potentially  significant 
and  were  used  to  supplement  the  interpretation  of  the 
experimental  results. 
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D.  Discussion  of  Results 

Preparation  of  Request 

It  Is  apparent  from  the  results  that  all  groups 
were  the  least  satisified  with  this  phase  of  the  CIS  on-line 
system.  The  most  notable  example  of  this  comes  from  Table  II, 
where  a  higher  level  of  knowledge  of  document  reference  and 
retrieval  systems  resulted  in  by  far  the  lowest  rating  of  any 
phase  of  the  search  procedure.  In  contrast,  the  groups 
representing  a  lower  level  of  experience  rated  the  preparation 
of  request  phase  much  higher.  One  likely  explanation  is  that 
knowledgeable  people  were  more  critical  because  they  were 
aware  of  alternative  methods  which  might  prove  more 
satisfactory  than  the  existing  method. 

Although  any  results  pertaining  to  times  derived 
from  this  experiment  not  definitive,  Table  III  provides  some 
interesting  figures  in  regard  to  request  formulation.  When 
Groups  I  and  III  are  compared  to  Groups  II  and  IV,  it  is 
noted  that  a  previous  knowledge  of  document  reference  and 
retrieval  systems  resulted  in  a  reduction  of  approximately 
one-half  the  time  required  to  formulate  a  request.  This  Is 
probably  most  accurately  attributed  to  a  familiarity  with  the 
vocabulary  used  by  the  system. 

In  attempting  to  pinpoint  the  main  areas  of  dis¬ 
content  with  this  phase  of  the  search  procedure,  the  comments 
from  the  questionnaires  are  particularly  interesting.  The 
comments  most  frequently  cited  are  the  preference  of  natural 
language  over  numbers  for  index  terms,  the  lack  of  a  logical 
organization  of  the  thesaurus,  the  inadequacy  of  the  scope 
notes,  and  the  fact  that  the  current  arrangements  for  using 
the  system  (scheduling  in  advance)  would  be  unsatisfactory  if 
the  system  were  operational.  As  might  be  expected  after 
examining  the  ratings  in  Tables  I  and  II,  the  portion  of  the 
questionnaire  dealing  with  request  preparation  was  the 
recipient  of  the  most  critical  comments. 


. .  .'KOirnFH'*  '"I'!-: 'HI  ill  M 


39  - 


i 

t 

I 

r 

% 

t 

i 

I  . 


j  ‘ 
\ 

jt 


f 

i 

i 

i 


I 


It  should  be  mentioned  that,  with  few  exceptions, 
all  Initial  requests  formulated  were  too  specific  for  the 
conditions  of  the  experiment.  That  Is,  the  subjects 
retrieved  less  than  five  documents.  The  exceptions  were  all 
submitted  by  subjects  from  the  higher  level  of  knowledge  of 
document  reference  and  retrieval  systems. 

Submission  of  Request 

For  the  two  levels  of  experience  with  on-line 
systems,  Table  I  shows  that  the  largest  difference  in  ratings 
occurred  in  this  phase  of  the  search  procedure.  In 
addition,  only  the  groups  having  little  or  no  previous 
experience  with  on-line  systems  rated  this  below  the  group 
average.  These  two  facts  could  serve  to  indicate  that  the 
terminal  Is  cumbersome  to  work  with  until  the  user  becomes 
familiar  with  It. 

This  conjecture  is  borne  out  In  part  by  the  comments 
appearing  on  the  questionnaires.  The  main  complaints 
suggested  that  the  terminal  was  confusing  to  operate  and  tha- 
the  routine  to  analyze  the  request  was  too  sensitive  to 
errors.  Just  as  common  were  the  favorable  comments  that  the 
conversation  routine  nade  the  system  easy  to  use  and  that  the 
program  performance  was  exceptionally  fast  for  the  computer 
being  used. 

Initial  Search 

This  phase  of  the  search  procedure  received  the 
highest  rating  In  three  of  the  four  groupings  shown  In  Tables 
I  and  II.  A  deeper  analysis  of  the  questionnaires  received 
indicated  that  all  three  system  components  being  Judged, 
hardware,  software,  and  the  system  itself,  were  evaluated  about 
equally  by  the  subjects. 

Most  of  the  coranents  centered  on  the  associative 
search  capabilities  of  the  system.  Although  it  was  general?y 
credited  with  being  a  good  approach  for  learning  about  the 
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way  index  terms  are  used  in  the  system,  it  was  also  noted 
that  an  associativity  table  is  of  little  value  when  it  is 
based  on  a  small  number  of  documents. 

Retrieval 

In  general,  there  was  little  disagreement  among  the 
various  groups  in  their  evaluation  of  this  phase  of  the 
search  procedure.  The  majority  of  the  comments  referred  to 
the  time-consuming  print-out  of  the  options  available  for 
presentation.  It  was  also  commonly  noted  that,  although  the 
document  collection  coverage  was  adequate  to  test  the  concept, 
it  was  too  limited  for  an  operational  environment. 

Reformulation  of  Request 

Surprisingly  little  difference  in  rankings  occurred 
in  the  evaluation  of  this  phase  of  the  search  procedure. 

Since  a  certain  amount  of  thought  is  required  to  successfully 
reformulate  a  request,  this  was  interpreted  as  a  healthy 
indicator  that  a  user  is  not  hindered  in  the  use  of  the  on¬ 
line  system  by  his  lack  of  experience. 

The  comments  Indicated  that  associativity  was  a 
useful  tool  in  the  reformulation  of  the  initial  request,  but 
was  not  always  adequate  in  locating  alternative  index  terms. 
Some  confusion  about  the  associativity  of  related  terms  was 
mentioned. 

Termination  of  Search 

This  phase  of  the  search  procedure  was  included  in 
the  questionnaire  primarily  for  the  sake  of  completeness  and 
resulted  in  rankings  near  the  group  average  and  relatively 
few  comments.  In  fact,  the  only  comment  generated  mentioned 
that  the  conversation  routine  was  easy  to  understand. 
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The  majority  of  the  annoyances  indicated  on  the 
questionnaire  dealt  with  the  system  itself  and  less  so  with 
the  associated  hardware  and  software  components.  It  is 
significant  to  point  out  that  most  of  these  annoyances  can  be 
easily  remedied  in  an  operational  system  and  the  remainder  of 
them  tend  to  disappear  as  the  familiarity  of  the  user  with 
the  system  increases.  The  major  complaints,  not  unexpectedly, 
centered  on  the  vocabulary  control  employed  by  the  system — 
the  use  of  term  numbers  rather  than  natural  language  and  the 
absence  of  a  logical  classification  of  the  terms  in  the 
thesaurus.  The  need  for  a  more  sophisticated  operating 
environment  wa3  singled  out  as  a  prerequisite  for  expansion 
of  the  system  beyond  the  experimental  stage.  An  unexpected 
result  was  the  apparent  heavy  reliance  upon  scope  notes  in 
formulating  a  request.  Similarly,  the  complaint  that  the 
numerical  format  for  request  formulation  was  too  stringent 
was  not  anticipated. 
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V,  Conclusions 

Generally  speaking,  the  user  reaction  to  the  CIS 
on-line  document  reference  and  retrieval  system  was  favorable, 
based  on  the  results  of  this  study.  Little  difference 
was  noted  in  the  average  rating  given  on  the  questionnaire, 
regardless  of  the  user's  knowledge  of  document  reference 
and  retrieval  systems  or  his  level  of  experience  with  on¬ 
line  systems.  All  groups  tested  rated  the  system  near  the 
"satisfactory"  range,  and  relatively  few  "severe  problems" 
were  mentioned.  It  would  appear  from  these  results  that, 
from  the  standpoint  of  a  user,  the  CIS  system  has  been 
successful  in  providing  man-machine  interaction  with  the 
contents  of  a  coordinate  indexed  document  collection. 

It  is  important  to  understand  that  system  conversion 
from  a  batch-processing  document  reference  system  to  an  on¬ 
line  mode  requires  a  system  redesign  to  maximize  benefits. 

The  magnitude  of  the  conversion  factors  to  achieve  user- 
systea  interface  and  expanded  service  is  too  great  for  the 
considerations  of  a  direct  conversion  of  an  existing  batch¬ 
processing  system.  That  is,  moving  from  a  tape-oriented 
to  an  on-line  system  requires  too  many  basic  design  decisions 
to  allow  mere  amplification  of  earlier  system.  The  logic 
of  the  tape-oriented  search  software  requires  modification. 

New  techniques  of  file  searching  and  file  maintenance  must 
be  developed.  New  programming  systems  of  scheduling,  main- 
tensunao  diagnostics,  and  communication  must  be  established. 

Because  conversion  to  on-line  systems  is  a  redesign 
function,  the  costs  for  conversion  are  high  in  relation  to 
design  of  the  initial  system.  Over  half  of  the  costs  are 
those  for  the  designing  and  programming  of  the  system's 
software.  The  level  of  these  costs  will  depend  on  the 


-  as  - 


desired  system  sophistication.  Communications  oriented 
terminals  and  computer  hardware  are  necessary  to  support 
the  system. 

The  potential  benefits  of  an  interactive  system  with 
associativity  searches  are  substantial,  in  that  access  is 
provided  to  a  document  collection  without  the  aid  of  a 
human  intermediary.  The  direct  access  capability  of  disk 
storage  provides  for  immediate  retrieval  of  document 
representations  in  response  to  the  request  of  the  user. 
Equally  important,  the  storage  requirements  are  not  pro¬ 
hibitive.  The  concept  of  associativity  is  valuable  in 
adapting  to  the  needs  of  both  the  uninitiated  and  regular 
user.  It  provides  information  about  the  system  vocabulary 
which  can  be  used  to  formulate  a  request  or  to  adjust  the 
volume  of  output  generated  by  the  initial  request.  Lastly, 
the  techniques  used  in  this  system  can  be  employed  in  any 
application  dealing  with  documents  and  the  assignment  of 
descriptors  to  those  documents  through  a  coordinate  index¬ 
ing  approach. 


APPENDIX  I 

EXAMPLE  OP  SEARCH 


An  example  of  on-line  inquiry  and  search  teletype  out¬ 
put  using  the  CIS  Document  and  Reference  Retrieval  System  is 
covered  on  the  next  two  panes.  It  illustrates  features  im¬ 
plemented  in  these  studies  and  discussed  in  the  text  of  this 
report.  It  can  be  contrasted  with  earlier  examples  in  the 
Systems  Manual  of  April  196?. 


THIS  IS  THE  CIS  DOCUMENT  REFERENCE  SEARCH  SYSTEM 


DO  YOU  WANT  AN  ASSOCIATIVE  SEARCH 
?  YES 


PLEASE  TYRE  YOUR  SEARCH  STATEMENT. 
?<52800V90700>*<72500Vb2200> 


THE  TERMS  REPRESENTED  IN  YOUR  SEARCH  STATEMENT  ARE 

52800  -  AUTOMATA 

90700  -  TURING  MACHINE 

72500  “  LIST  PROCESSING 

52200  -  ARTIFICIAL  INTELLIGENCE 


ARE  THESE  TERMS  0-K.  WITH  YOU 
?  YES 


YOUR  REQUEST  YIELDS  4  DOCUMENTS. 


TERM  ASSOCIATIONS 


TERM  NO 

TOT.  OCC. 

CO-OCC. 

TERM 

ASSOC I  ATI  VI 'I'Y 

52200 

54 

A 

ARTIFICIAL  INTELLIGENCE 

W  •  0  7  A  1 

5280M 

17 

2 

AUTOMATA 

0.0588 

64500 

10 

1 

GAME 

0. 0250 

7  1500 

36 

A 

LEARNING 

0.1111 

72900 

92 

3 

MACHINE 

0.0245 

88600 

1  1 

l 

TEACHING 

0.0227 

89600 

10 

1 

THINKING 

0.0250 

90700 

7 

2 

TURING  MACHINE 

0. 1 429 

DO  YOU  WANT  TO  REFORMULATE  YOUR  CURRENT  SEARCH  STATEMENT 
7  NO 


PLEASE  SELECT  ONE  OF  THE  FOLLOWING  OUTPUT  OPTIONS.  INDICATE  YOUR 
CHOICE  8Y  TYPING  THE  APPROPRIATE  NUMBER. 

(1)  DOCUMENT  ACCESSION  NUMBER  ONLY. 

(2)  DOCUMENT  ACCESSION  NUMBER.  AUTHOR.  AND  TITLE* 

<3>  ALL  OF  THE  ABOVE  WITH  BIBLIOGRAPHY. 

7  3 


1-1 


CIS  DOC.  NO.  F  891 

KOCHKN.  M.  A  MODEL  FOR  THE  PROCESS  OF  LEARNING  TO  COMPREHEND 

IBM  RES  CENTER#  TO RK TOWN  HEIGHTS.  NY.  AFCRL-64-87.  AF  19<638>-27b2 
APR  1964 


CIS  DOC.  NO.  B  1230 

TOO.  J.  AND  R.  WILCOX  EDS*  COMPUTER  AND  INFO  SCIENCES 
SPARTAN  BOOKS  INC.  WASHINGTON 


CIS  DOC.  NO.  8  1984 

TURING.  A.  COMPUTING  MACHINERY  AND  INTELLIGENCE 

IN  FEIGEN8AUM.E.  AND  J.  FELDMAN. EDS.  COMPUTERS  AND  THOUGHT . MCGRAW 
HILL.  1-35 


CIS  DOC.  NO.  8  1985 

NEWELL. A.  SHAW.  J •  SlMON.H.  CHESS-PLAYING  PROGRAMS  AND  THE  PROBLEM  OF 
COMPLEXITY 

IN  FEIGENBAUM.E.  AND  J. FELDMAN.  EDS.  COMPUTERS  AND  THOUGHT.  MCGRAW 
HILL.  39-70 


DO  YOU  WANT  TO  ENTER  ANY  MORE  SEARCH  STATEMENTS 

?  N  P 


GOOD-BYE 


COME  BACK  AGAIN 


APPENDIX  II 


Cantor  for  the  Information  Sciences 
Lehigh  University 
Bethlehem,  Pa.  18015 


You  are  being  asked  to  evaluate  the  CIS  experimental  document 
reference  and  retrieval  system  on  the  basis  of  the  following  com¬ 
ponents,  Included  below  are  suggestions  to  help  formulate  your 
evaluation  or  generate  comments, 

I,  System 

A,  Vocabulary  oontrol 

1,  index  terms  in  the  thesaurus 

2,  use  of  term  numbers  for  searching  (rather  than 
natural  language) 

3*  index  term  qualifiers,  or  scope  notes 
4*  coordinate  indexing  approach 
B«  Search  logic 

1,  Boolean  operators  used  (+,  V,  -) 

2.  four  term  limit 

C.  Conversation  routine 

1.  order  in  which  questions  a1^-  prest-nted 

2.  adequaoy  of  questions 

3.  formats 

D.  Document  collection 

1.  ooverage 

2,  quality  of  documents  retrieved 

E.  Indexing 

1«  consistency 
2.  too  heavy  or  too  light 
P.  Associativity 

1.  as  tool  for  search  reformulation 

2.  relatedness  of  query  terms  and  other  index 
terms  (Stiles'  formula) 

3.  quantity  of  terms  displayed 


II-2 


II,  Hardware 

A.  Terminal 

1,  display  purposes 

2,  keyboard  for  Input 

3,  speed 

B.  Communication  line 

1,  time  between  placing  initial  call  and  beginning 
of  program  type -out 

2,  hardware  interruptions  during  search  procedures 
III*  Software 

A.  Access  to  system 

1.  waiting  time  to  use  system 

2.  turnaround  time 

B.  Response  time 

1,  time  between  entering  request  and  learning  number 
of  documents  which  satisfy  request 

2.  time  between  learning  number  of  documents  which 
satisfy  request  and  print-out  of  document  repre¬ 
sentations 

C.  Program  performance 

1.  failure  to  perform  as  designed 

2.  design  limitations 
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APPENDIX  IV 
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II  ABSTRACT 


An  existing  experimental  document  and  reference  retrieval  system 
operating  in  batch  processing,  tape  oriented  mode  was  converted 
to  an  on-line  mode  with  user  interaction.  Objectives  were  faster 
response  time,  integration  of  functions,  and  system  accessibility 
at  user  location.  Effects  on  file  organization,  addressing  tech¬ 
niques.  file  maintenance,  communication,  scheduling  and  programming 
are  analyzed.  Modification  of  an  existing  batch  oriented  system  can 
yield  an  effective  interactive  search  capability.  Realization  of 
full  benefits  of  the  on-line  concept,  particularly  maintenance  of  the 
document  base,  requires  system  redesign  rather  than  extension  of  ex¬ 
isting  procedures. 

User  reaction  to  the  experimental  system  was  favorable  although  a 
variety  of  improvements  were  suggested  for  an  operational  environ¬ 
ment.  Output  of  associativity  indicators  was  valuable  in  reformu¬ 
lating  search  requests  or  in  adjusting  the  volume  of  cited  references 
for  both  experienced  and  unitiated  users. 
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